System and method for digital markups of custom products

ABSTRACT

Techniques for generating and using digital markups on digital images are presented. In an embodiment, a method comprises receiving, at an electronic device, a digital layout image that represents a form of a product for manufacturing a reference product; generating a digital markup layout by overlaying the digital markup image over the digital layout image; based on the digital markup layout, generating one or more manufacturing files comprising digital data for manufacturing the reference product; receiving a digital reference image of the reference product manufactured based on the one or more manufacturing files; identifying one or more found markup regions in the digital reference image; based on the found markup regions, generating a geometry map and an interactive asset image; based on, at least in part, the geometry map, generating a customized product image by applying a user pattern to the interactive asset image.

This application claims the benefit under 35 U.S.C. § 120 as acontinuation of application Ser. No. 16/791,977, filed Feb. 14, 2020,which is a continuation of U.S. patent application Ser. No. 16/230,678,filed Dec. 21, 2018, now U.S. Pat. No. 10,580,185, which is acontinuation of U.S. patent application Ser. No. 15/818,514, filed Nov.20, 2017, now U.S. Pat. No. 10,176,617, which is a continuation of U.S.patent application Ser. No. 15/232,592, filed Aug. 9, 2016, now U.S.Pat. No. 9,852,533, which claims the benefit under 35 U.S.C. 119(e) ofprovisional application 62/203,026, filed Aug. 10, 2015, the entirecontents of which are hereby incorporated by reference for all purposesas if fully set forth herein. The applicant hereby rescinds anydisclaimer of claim scope in the parent applications or the prosecutionhistory thereof and advises the USPTO that the claims in thisapplication may be broader than any claim in the parent applications.

FIELD OF THE DISCLOSURE

The present invention relates to computer-implemented techniques forusing digital markups on digital images for purposes of recognizingsurfaces of deformable objects depicted in the digital images and usingthe recognized surfaces to apply customized patterns to the deformableobjects depicted in the digital images.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Consumer shopping networks and websites form a big sector of the retailmarket. They strive to provide convenience to consumers; however, theyoften fall short in providing a true visual representation of customizedproducts. For example, sometimes they are unable to realistically depictvarious patterns, textures or finishes on digitally customized productssuch as wearable apparel, toys, furniture, stationeries, and the like.

To increase the realism of customized products displayed digitally, somecompanies implement markups applied to digital images of the customizedproducts. When a markup is well-described and can be clearly seen on adigitally displayed product, the markup may be easily identified, andthus additional customization of a digital image of the product may beautomated. However, when only a small portion of the markup is visibleon the displayed product, or the markup is deformed or folded in acomplex manner, the markup may be difficult to identify and processingof the markup using automated approaches may be challenging.

SUMMARY

The appended claims may serve as a summary of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates an example image processing computing system in whichan embodiment may be implemented.

FIG. 2 illustrates an example marker element of a digital markup image.

FIG. 3 illustrates an example digital markup image.

FIG. 4 illustrates example digital layout images.

FIG. 5 illustrates an example of overlaying a digital markup image overa digital layout image.

FIG. 6 illustrates an example design area that shows a digital markupimage overlaid over a digital layout image and that is displayed to auser in a graphical user interface.

FIG. 7 illustrates an example digital reference image.

FIG. 8 illustrates an example digital reference image to be processedusing a color filtering approach to determine a digital markup imprintedin the image.

FIG. 9 illustrates an example image having markups removed.

FIG. 10 illustrates an example customized product image that is renderedby applying a user pattern to an interactive asset image.

FIG. 11 illustrates an example digital markup image having a markuparrangement of dots that is unique for rotations by 0, 90, 180, 270degrees.

FIG. 12 illustrates an example process for using digital markups ondigital images for the purpose of recognizing markup regions in digitalreference images and applying customized patterns to deformable objects.

FIG. 13 illustrates an example process for identifying markups indigital reference images using a color filtering approach.

FIG. 14 is a block diagram that illustrates a computer system upon whichan embodiment of the invention may be implemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent approach.

Embodiments are described herein according to the following outline:

I. General Overview

-   -   A. Introduction    -   B. Overview

II. Structural and Functional Overview

-   -   A. Digital Markup Generator        -   1. Digital Markups with Checksums        -   2. Digital Markups with Unique Arrangements for Rotations        -   3. Digital Markup Identifiers    -   B. Digital Markup Storage

III. Applying a Digital Markup to a Digital Layout

-   -   A. Examples of Digital Layouts    -   B. Grid-Based Mapping

IV. Generating Manufacturing Files Containing Digital Markups

V. Generating a Digital Reference Image

VI. Segmenting Digital Markups in Digital Reference Images

-   -   A. Segmenting a Digital Markup Using a Color-Filtering Approach    -   B. Building a List of Markup Identifiers    -   C. Generating a Geometry Mapping

VII. Generating an Interactive Asset Image

VIII. Generating a Customized Product Image

IX. Implementation Example—Hardware Overview

I. General Overview A. Introduction

In an embodiment, techniques are described for generating digitalmarkups and using the digital markups to generate digitalrepresentations of customized products. A digital markup may be forexample, a digital image that has a unique pattern or a uniquearrangement of digital elements.

The techniques may include applying a digital markup to a digital image,using the marked-up digital image to manufacture a physical referenceproduct, and obtaining a reference image representing a photograph ofthe manufactured reference product and having the imprinted digitalmarkup. The techniques may also include recognizing surfaces of thedigital markup depicted in the reference image and applying a customizedpattern to the surfaces recognized in the reference image. Thetechniques may be used to process digital images of wearable apparel,toys, and other objects that have components or elements covered withpatterns or textures.

The techniques may be implemented in computer-based shopping websites,advertising websites, and other computer-based applications thatdigitally visualize various products offered in various patterns,textures or finishes. An example implementation may include a websitefeaturing a particular piece of clothing that is available in threedifferent versions: a version with a polka-dotted pattern, a versionwith a plaid pattern, and a version with a zig-zag pattern.Traditionally, to realistically represent the three versions of digitalmedia, a website developer would take three pictures of differentmannequins, each mannequin wearing a different version of the particularpiece of clothing and post the three pictures on the website for usersto see. Each of the pictures would realistically represent the folds ofthe clothing and the folds of the patterns on the clothing. However,obtaining three (or more) different pieces of clothing for each piece ofclothing that the website features, and taking three (or more) differentpictures for each and every featured piece may be time consuming andexpensive. But the presented techniques require obtaining only one pieceof clothing and only one picture of the piece. The additional picturesmay be generated by applying different patterns to the same picture insuch a way that the same deformations of the patterns are depicted oneach of the customized images.

In an embodiment, this is accomplished by using digital markups tocut-out forms used by a manufacturer. For example, a digital markup maybe applied to a cut-out form used by a manufacturer to produce aparticular piece of clothing. Upon receiving a digital image of apicture of a mannequin wearing the marked-up piece of clothing, thedigital image may be modified by replacing the markup depicted on thedigital image with a customized pattern. The replacement may beperformed according to the digital markup depicted on the digital image.Therefore, instead of generating separate pictures of each individualvariation of the piece of clothing, only one physical piece and onepicture of the piece is needed. The additional pictures showing the samepiece of clothing, but having different patterns, textures or finishes,may be generated automatically.

Since it is important to generate a realistic representation of thepiece of clothing as it was featured on a mannequin or a person, adigital representation of the piece needs to capture the fact that thepiece may be deformed and wrapped around the model's body. In anembodiment, the presented approach allows the capture of thedeformations and the wrapping of the piece and allows to realisticallyreproduce the corresponding deformations of the customized patternsdepicted on the customized image of the piece.

B. Overview

In an embodiment, a computer-implemented method comprises receiving, atan electronic device, a digital markup image. The digital markup imagemay have one or more reference markup regions and may be associated witha markup identifier. The digital markup image may represent a digitalmarkup that has one or more unique patterns arranged in a unique way ora composition.

In an embodiment, a computer-implemented method comprises receiving adigital layout image that represents a cut-out used to manufacture areference product. Examples of reference products may include a piece ofclothing, a soft toy, a piece of furniture having a cloth upholstery,and the like.

Based on the digital markup image and the digital layout image, adigital markup layout may be generated by overlaying the digital markupimage over the digital layout image. This may be performed using animage processing tool that allows projecting the digital markup imageonto the digital layout image in such a way so that both images can beseen in the resulting digital markup layout.

In an embodiment, based on, at least in part, the digital markup layout,one or more manufacturing files are generated. The manufacturing filesmay comprise digital data usable in manufacturing a physical product,also referred to as a reference product. The manufacturing files may berepresented in any format that a computer system of the manufacturer canaccept and use to control a cutting of a fabric or cloth to formphysical pieces of the reference product. Once the reference product ismanufactured, a picture is taken of the reference product as it is wornby a model, a mannequin, or used by customers. A digital form of thepicture is referred to as a digital reference image.

In an embodiment, a digital reference image of the reference product isreceived. The digital reference image may be for example, a digitalphotograph of a model wearing the reference product manufactured basedon the manufacturing files.

Since the reference product, such as a pair of leggings, may appear onthe digital reference image as wrapped around the model's body, thedigital markup may appear on the digital reference image deformed orcropped. Furthermore, the digital reference image of the referenceproduct may include only a portion of the digital markup image.Therefore, recognizing the digital markup just by visually inspectingthe digital reference image may be difficult and not straightforward.

In an embodiment, a digital reference image of a reference product isdigitally analyzed, and one or more markup regions, also referred to asfound markup regions, are identified in the digital reference image. Thefound markup regions are the regions that are suspected to correspond tosome regions of a digital markup image. Identifying the found markupregions may be performed using various approaches such as imageprocessing and shape recognition techniques described in detail below.

In an embodiment, one or more reference markup regions of a digitalmarkup image used to generate the digital markup layout are retrieved.The one or more reference markup regions are used to determine a regionmapping that maps one or more reference markup regions of the digitalmarkup image to one or more found markup regions identified in thedigital reference image.

A region mapping is a mapping that maps some of the reference markupregions of the digital markup image onto some of the found markupregions identified in the digital reference image. As described above,since the reference product, such as a pair of leggings, may appear onthe digital reference image deformed or cropped, it is possible thatonly some regions of a digital markup image may be found.

In an embodiment, based on, at least in part, a region mapping, ageometry map is generated. A geometry map may comprise a plurality ofpolygons generated based on found markup regions. Each of the polygonsmay correspond to a different portion of the digital markup image, andeach of the polygons may capture a deformation of the portion of thedigital markup.

In an embodiment, an interactive asset image is generated from a digitalreference image. This may be performed by removing markings from thedigital reference image and obtaining an unmarked version of the digitalreference image. The unmarked version of the digital reference image maybe then customized using customized patterns, such as a user pattern, apattern designed by a manufacturer, or a pattern designed by a webdeveloper. This may include retrieving the customized pattern from adata storage device or a server and applying the customized pattern tothe unmarked version of the digital reference image to generate acustomized product image. The customized pattern may be applied in sucha way that the customized pattern appears to be deformed in the samefashion as the markup regions were deformed on the digital referenceimage.

In an embodiment, a customized product image is displayed on a displaydevice or other display media. For example, the customized product imagemay be sent to a customer device to cause displaying the customizedproduct image on a display of the customer device. According to anotherexample, the customized product image may be used to generate a newwebpage of a website, and the new webpage may be posted on the website.According to another example, the customized product image may be sentto a smartphone operated by a user, and the customized product image maybe displayed on a display of the smartphone.

In an embodiment, techniques for using digital markups on digital imagesis implemented using a series of ordered process steps. The steps areintended to illustrate example algorithms that may be implemented usingone or more computers.

In an embodiment, techniques for using digital markups on digital imagesfor purposes of recognizing surfaces and applying customized patterns todeformable objects such as wearing apparel are implemented using anytype of electronic devices, such as a computer, a workstation, a laptop,a server, a smartphone and other electronic device configured toreceive, process and transmit digital images.

In an embodiment, the steps describe an algorithm, process, or otheroutline for how the functions are implemented in programminginstructions for a computer. Any suitable programming language ordevelopment environment may be used such as JAVA, OBJECTIVE-C, C++,scripting languages, and the like. In practice, an implementation orembodiment will include program instructions for many steps other thanthose listed herein, but the specific listings of steps neverthelessindicate the algorithmic information sufficient to communicate to askilled programmer or software engineer how to implement the specifiedfunctions in complete working code.

In an embodiment, techniques are implemented in a digital markercomputer processing (“DMCP”) system. The DMCP may generate digitalmarkers that have certain characteristics that provide informationneeded to solve a task of mapping markup on a variety of manufacturedsurfaces, such as apparel. In the DMCP, the pattern of the digitalmarkup itself can provide or improve the marker coordinate frame, andthe pattern itself may characterize a local surface deformation orcurvature.

In one embodiment, the steps to build, apply and recognize the DMCPcomprise defining digital markups, defining digital markup identifiers,applying a digital markup to a digital layout and generating one or moremanufacturing files containing digital markups. Once a reference productis manufactured based on the manufacturing files, a region mapping, anda geometry mapping are generated. The mappings are used to generate aninteractive asset image, which is then used to generate a customizedproduct image by applying a customized pattern to the interactive assetimage.

II. Structural and Functional Overview

FIG. 1 illustrates an example image processing computing system in whichan embodiment may be implemented. In an embodiment, an example imageprocessing computing system 102 is implemented in a computingenvironment 100. Image processing computing system 102 may be configuredto generate digital markups and use the digital markups to generatedigital representations of customized products. Image processingcomputing system 102 may generate a plurality of digital markup images,each digital markup image having a unique pattern or a uniquearrangement of digital elements.

In an embodiment, image processing computing system 102 is configured toapply a digital markup to a digital image, use the marked-up digitalimage to manufacture a physical reference product, obtain a referenceimage representing a photograph of the manufactured reference productand having the imprinted digital markup, recognize surfaces of thedigital markup depicted in the reference image, and apply a customizedpattern to the surfaces recognized in the reference image.

In an embodiment, image processing computing system 102 comprises one ormore computer processors 101 configured to execute program instructionsstored in one or more memory units 103. Image processing computingsystem 102 may also include one or more of: a digital markup generator104, a layout manager 106, a digital markup applicator 108,manufacturing file generator 110, a reference product manager 112, oneor more memory units 103, a markup segmenting unit 114, a markup regionsgenerator 116, a markup recognizer 118, a geometry map builder 120, oran interactive asset builder 122.

In an embodiment, digital markup generator 104 is programmed orconfigured to generate a plurality of digital markup images. A digitalmarkup image may be any type of a digital image that has a uniquepattern or a unique arrangement of digital elements.

In an embodiment, digital markup generator 104 is programmed orconfigured to determine a checksum for a digital markup image. Digitalmarkup generator 104 may also be configured to generate a plurality ofdigital pattern images that have the checksum embedded in each digitalpattern image. A checksum may be a count of rows (and columns) in anindividual digital pattern of a digital markup image. Based on theplurality of digital pattern images, digital markup generator 104 maygenerate a plurality of rotated digital pattern images, select a subsetof the plurality of rotated digital pattern images, and generate thedigital markup image based on the subset of the plurality of rotateddigital pattern images.

Digital markup generator 104 may also be programmed or configured togenerate a markup identifier for a digital markup image. A markupidentifier may be generated based on the contents of the digital markupimage. A markup identified may be stored along with the digital markupimage in a markup index data structure.

In an embodiment, layout manager 106 is programmed or configured toreceive a digital markup image that has one or more reference markupregions and is associated with a markup identifier. Layout manager 106may also be configured to receive a digital layout image that representsa form of a product used to manufacture a reference product.

In an embodiment, digital markup applicator 108 is programmed orconfigured to generate a digital markup layout by overlaying the digitalmarkup image over the digital layout image. This may be performed usingan image processing tool that allows overlaying the digital markup imageover the digital layout image so that both images can be seen in theresulting digital markup layout.

In an embodiment, manufacturing file generator 110 is programmed orconfigured to generate one or more manufacturing files based on, atleast in part, a digital markup layout. The one or more manufacturingfiles may comprise digital data usable in manufacturing the referenceproduct. For example, manufacturing file generator 110 may be programmedor configured to scan a digital markup layout to generate a scanned fileand modify the scanned file by adding a program instructing for usingthe scanned digital markup layout to generate physical components of thephysical reference product.

A scanned file may be sent to a manufacturer and downloaded on acomputer system operated by the manufacturer. Upon receiving the scannedfile, the computer system operated by the manufacturer may initiategenerating one or more physical components of a physical referenceproduct. The component then may be assembled into the reference product.

In an embodiment, reference product manager 112 is programmed orconfigured to receive a digital reference image of a physical referenceproduct that has been manufactured based on, at least in part, one ormore manufacturing files.

In an embodiment, markup segmenting unit 114 is programmed or configuredto determine a plurality of distinct regions in a digital layout imageand determine whether any of the distinct regions of the plurality ofdistinct regions contains at least a portion of a digital markup image.

Markup segmenting unit 114 may also be programmed or configured tosegment a digital reference image into one or more found markup regionsby performing one or more of: applying a color shed approach to thedigital reference image, determining distance values between colorvalues in areas of interest in the digital reference image and colorvalues in a reference photograph, generating a spatial frequencyhistogram for the digital reference image, or generating a bandpasshistogram for the digital reference image.

In an embodiment, markup regions generator 116 is programmed orconfigured to identify, based on a plurality of distinct regions, one ormore found markup regions in a digital reference image that potentiallyinclude a digital markup image.

In an embodiment, markup recognizer 118 is programmed or configured todetermine a region mapping that maps one or more reference markupregions to one or more found markup regions.

In an embodiment, geometry map builder 120 is programmed or configuredto generate a geometry map based on, at least in part, a region mapping.

In an embodiment, interactive asset builder 122 is programmed orconfigured to generate an interactive asset image by removing marking inone or more found markup regions of a digital reference image.

Interactive asset builder 122 may also be programmed or configured togenerate a customized product image based on, at least in part, a regionmapping and a geometry map. A customized product image may be generatedby applying a user pattern to an interactive asset image. Interactiveasset builder 122 may also be configured to cause the customized productimage to be displayed on a display device.

Besides image processing computing system 102, computing environment 100may also include one or more storage devices 103 accessible to the imageprocessing system 102.

Computing environment 100 may also include a digital camera 111configured to capture digital images of reference products and providethe digital images to image processing computer system 102.

Computing environment 100 may also include one or more user computingsystems 107 that may communicate with image processing computing system102 via a communication computer network 120. A user computing system107 may be configured to store images representing patterns, textures,and finishes in internal storages 172 and/or external storage devices,such as a storage device 173.

Computing environment 100 may also include one or more manufacturercomputers that may have one or more manufacturing file processors 124.Manufacturing file processors 134 may be configured to receivemanufacturing files from image processing computing system 102 and usethe manufacturing files to control manufacturing equipment tomanufacture physical parts of a physical reference product 142 and toassemble the parts into reference product 142. The manufacturingequipment may be maintained by a manufacturer 140.

Computing environment 100 may also include shipping services 150 thatallow receiving reference product 142 from manufacturer 140, and ship,or otherwise deliver, reference product 142 to image processingcomputing system 102. Upon receiving reference product 142, a camera 111may be used to take a picture of reference product 142, and a digitalreference image of the reference product may be provided to referenceproduct manager 112.

A. Digital Markup Generator

Digital markup generator 104 is configured to generate digital markups.Digital markups are generally applicable to any kind of printabledeformable surface, including all kinds of clothing, shoes, textilessuch as bedding, plush toys, and the like.

For purposes of illustrating an example of a digital marker, certainembodiments are explained with reference to wearing apparel, such as apair of leggings; however, the disclosure is not limited toimplementations for the wearing apparel. A pair of leggings is a type ofapparel worn on the legs and hips. Due to its structure and purpose, theleggings wrap around the legs and hips. The wrapping may be representedor captured using a deformable surface. In an embodiment, a digitalmarker may be placed on any type of a surface, including a deformablesurface. Using the presented approach, the coordinates and deformationparameters of the marker may be found for any type of the surface.

A digital markup may be applied to cut-outs or layouts of clothingmaterials used to assemble a piece of clothing. Once the piece ofclothing is assembled, the piece of clothing may have the digital markupimprinted on it. The piece of clothing may be put on a mannequin or amodel and photographed using a camera. The photograph may be thenprocessed to recognize the digital markup on the photograph, and therecognized markup may be used to determine a geometry map of the surfaceon which the markup is present. The geometry may be used to generate anunmarked digital image, and a custom pattern may be applied to theunmarked digital image to generate an interactive asset depicting thepiece of clothing having the customized pattern.

In an embodiment, a digital markup is a digital markup image that hascertain properties that distinguish the digital markup from otherdigital markups. A digital markup may comprise one or more digitalpatterns, also referred to as elements. Each digital pattern may beencoded as an array of bits in a regular pattern that has even intervalsbetween elements. For example, a digital pattern may be encoded as a setof 4×4 dots. If a digital pattern is encoded using a set of 4×4 dots,then the total number of possible patterns is 16.

FIG. 2 illustrates an example marker element of a digital markup image.In the depicted example, a marker element has four rows and four columnsstoring dots that are either large dots or small dots. For example, dots210 and 220 are small dots, while a dot 230 is a large dot. The largedots may correspond to a bit set to ‘true,’ while the small dots maycorrespond to a bit set to ‘false.’

In an embodiment, a digital pattern or element may be represented by asequence of bits corresponding to dots depicted in the digital pattern.A large dot may be represented by ‘one’ or ‘true.’ A small dot may berepresented by ‘zero’ or ‘false.” Hence, a bit may be set to ‘true’ bymaking the dot larger, or ‘false’ by making the dot smaller. The sizechange may be determined by the best recognizable separation betweenelements, and the best recognizable scale for determining a ‘true’ stateor a ‘false’ state.

In an embodiment, a digital pattern may be represented by a value of avector that represents the pattern. The value of an encoded binarynumber in the markup is limited by 0 . . . 2^(n)

In embodiment, a count of elements in a digital pattern set to ‘true’may be limited to n/2 to embed a checksum in the markup.

Binary numbers encoded in a markup that match a number with a lowervalue if its array of elements is rotated by 90 degrees are used toembed rotation in the markup, rather than as separate markup numbers.

FIG. 3 illustrates an example digital markup image. The depicted digitalmarkup image 310 includes a plurality of digital patterns arranged in atable having a certain number of rows and columns. The digital patternsare selected in such a way that non two digital patterns from theplurality of digital patterns are identical. Since no two digitalpatterns in the plurality of digital patterns in the digital markupimage are identical, the arrangement of the plurality of digitalpatterns in the digital markup image is unique.

1. Digital Markups with Checksums

In an embodiment, a digital markup image has a checksum. A digitalchecksum is the number of large dots in a 4×4 pattern. For instance, achecksum of 8 would mean that the digital markup is that subset of 4×4patterns that have 8 large dots and 8 small dots.

In an embodiment, a checksum is determined before a digital markup isgenerated and it is used as a parameter for generating the digitalmarkup image. A checksum need not be actually represented by a numberincluded in the digital markup image. Instead, a checksum is known forthe digital markup image before the image is generated.

2. Digital Markups with Unique Arrangements for Rotations

In an embodiment, a digital markup image is created in such a way that amarkup pattern is distributed across individual markers.

FIG. 11 illustrates an example digital markup image having a markuparrangement of dots that is unique for rotations by 0, 90, 180, 270degrees. In the depicted example, a digital markup image comprises aplurality of digital markup patterns, wherein each digital markuppattern is a 4×4 element that has a unique arrangement of dots for eachof rotations of 0, 90, 180, and 270 degrees. A 4×4 element may berepresented as a grid having four squares in each of four rows. A squaremay include a dot or not. An example of a square with a dot is a square1120. Examples of squares without dots include squares 1110 and 1130.The pattern is designed so that no 4×4 sub-area is used in any other 4×4sub-area of the design in any of the 4 rotations. Therefore, a design of24×24 elements may use 21×21 or 441 unique overlapping 4×4 sub-designs.Using such arrangements reduces the image resolution required to encodethe design. Since the ability to generate a design having uniqueelements in each of rotations of 0, 90, 180 and 270 degrees, the sizelimit for the design with this kind of 4×4 uniqueness is 127×127.

3. Digital Markup Identifiers

In an embodiment, a markup has an associated markup identifier, which isreferred to as a MarkupID herein for convenience, although otherembodiments may use other designations. A markup identifier may begenerated based on a binary number corresponding to a value of thevector obtained from a markup pattern. One approach for generating amarkup identifier is to use a binary vector built based on the contentof the markup. For example, a digital markup pattern may be representedas a binary string, the binary string may be converted to a binaryvalue, the binary value may be used to generate a vector, and the vectormay be used as a MarkupID of the digital markup pattern.

In an embodiment, markup identifiers are generated based on ordinalvalues. The ordinal values may be generated and used to index a markupindex, also referred to as a MarkupIndex herein for convenience,although other embodiments may use other designations. For example, aMarkupID for a particular digital markup may be assigned the nextavailable ordinal value in a MarkupIndex.

The manner in which markup identifiers are created and used to index aMarkupIndex may be encoded as software code such as C++ code. Forexample, a C++ object may be created that includes instructions which,when executed, cause generating MarkupIDs and a MarkupIndex.

A process of creating digital markup identifiers for digital markups anddigital markup index may include digitally creating and storing, incomputer storage, a list of MarkupIDs based on the properties of thedigital markups. For example, a particular MarkupID for a particulardigital markup may be created based on either a value of the digitalmarkup vector or a next available ordinal value in a MarkupIndex.

The process may also include building a mapping from a MarkupIndex toMarkupIDs. This process may be performed in either direction. Forexample, if the MarkupIDs are created based on the markups' vectors,then the MarkupIDs may be used to generate the MarkupIndex, and then themapping. However, if a sequence of ordinary numbers is used to generatea MarkupIndex, then the ordinary numbers may be used to generate theMarkupIndex, and the ordinary numbers may be assigned to digital markupsas the MarkupIDs.

In an embodiment, a MarkupID associated with a markup includesinformation about rotations applicable to the markup image. For example,if a digital markup image is generated in such a way that its elementsare unique in a rotation by for example, 90 degrees, then the MarkupIDgenerated for the digital markup image may include not only theinformation about a markup pattern of the digital markup, but also theinformation about the rotation. Alternatively, the rotation informationmay be stored separately from the MarkupID. For example, the rotationinformation may be associated with the MarkupID.

In an embodiment, if a rotation value is included in a MarkupID for aparticular digital markup image, then a process of creating digitalmarkup identifiers includes a method for returning the rotation valuefor the particular digital markup. If the rotation value is a MarkupID,then returning value for the particular digital markup would includereturning the rotation value. However, if the rotation value is notincluded in the MarkupID, then two separate queries may be issued: oneto request a rotation value and another to request the MarkupID.

B. Digital Markup Storage

In an embodiment, digital markup images may be stored in a storagedevice, such as device 103 depicted in FIG. 1 . The storage device maystore the digital markup images using any type of data organization. Forexample, the digital markup images may be stored in association with aMarkupIndex and in association with any additional information if suchas available or provided.

III. Applying a Digital Markup to a Digital Layout

FIG. 12 illustrates an example process for using digital markups ondigital images for the purpose of recognizing markup regions in digitalreference images and applying customized patterns to deformable objects.

In step 1210, one or more digital markups are defined. Digital markupsmay be defined in advance. If two or more digital markups are defined,then the digital markups are defined in such a way that no two of thetwo or more digital markups are identical. Examples of digital markupsare described in FIG. 3 and FIG. 11 .

In an embodiment, a digital markup is defined by generating a uniquearrangement of dots of different sizes. The unique arrangement may berepresented in a digital markup image. A digital markup image may bestored using any data format, including the formats such as mpeg, jpeg,pix, and the like.

In step 1220, markup identifiers are assigned to digital markups.Assigning a markup identifier to a digital markup image may includegenerating a MarkupID based on a value of the vector representing thedigital markup image and assigning the value as the MarkupID of thedigital markup image.

Referring again to FIG. 12 , in step 1230, a digital markup image isapplied to a digital layout image. Applying a digital markup image to adigital layout image may be implemented using an automated approach thatautomatically fits the digital markup image to the digital layout imageor overlays the digital markup image over the digital layout image. Asdescribed above, a digital markup image may be a digital image thatrepresents a unique combination of dots, or squares, that have differentsizes, and each of the dots, or squares, represents either the binary‘1’ or the binary ‘0.’

A. Examples of Digital Layouts

A digital layout image may be a digital image representing a physicalcut-out or a physical form that a manufacturer may use to cut physicalpieces of for example, a cloth, a piece of garment, or a toy, that latermay be assembled into a physical product. Examples of digital layoutimages are described in FIG. 4 .

FIG. 4 illustrates example digital layout images. The depicted exampledigital layout images include two images: a digital layout image 430 anda digital layout image 450. Both images represent physical cut-out formsthat may be used by a manufacturer, or a seamstress, to cut out piecesof fabric to assemble, for example, a pair of leggings.

A cut-out form usually has certain markings imprinted on it. The certainmarkings may include several different types of lines, and a count ofthe different lines depicted in the cut-out form depends on thepreferences implemented by a manufacturer. For example, the markings mayrepresent leading lines for cutting the fabric, leading lines forstitching the fabric, and/or leading lines beyond which neither cuttingnor stitching could be performed.

In the example depicted in FIG. 4 , a digital layout image 430represents a cut-out form that has three types of markings. One marking,depicted using a broken line 410, indicates a cut-out line. This line isto be followed to cut a fabric. Another marking, depicted using a solidline 420, indicates a stitch line which is to be followed by a sewingmachine to assemble the garment. Other marking, depicted using a dashedline 440, indicates a safe line beyond which neither cutting norstitching should be performed.

B. Grid-Based Mapping

In an embodiment, a digital markup image is applied to a digital layoutimage provided by a manufacturer or a seamstress. Applying a digitalmarkup image to a digital layout image may be performed by overlayingthe digital markup image over the digital layout image in such a waythat an outline of the digital markup image follows an outline of thedigital layout image as much as possible. By overlaying a digital markupimage over a digital layout image, a digital markup layout is generated.

Since a digital markup image has usually a rectangular shape and adigital layout image rarely has a rectangular shape, overlaying thedigital markup image over the digital layout image in such a way thatthe corresponding outlines follow each other as much as possible mayinvolve stretching some areas of the digital markup image. Theoverlaying and the stretching may be performed using various techniques.One technique may involve dividing the digital overlay image into a gridand mapping the elements of the digital markup image onto the grididentified in the digital overlay image.

In an embodiment, a digital layout image is divided into a gridcomprising a plurality of grid elements. A grid element may have anytype of shape, not necessarily a rectangular shape. A size of the gridmay be determined based on a size of the digital markup image, aresolution of the manufacturing process, a size of the digital layoutimage, and/or a size of the physical reference product that will beassembled based on a cut-out form represented by the digital layoutimage. Elements of the grid are also referred to as distinct regions ofthe digital layout image.

In an embodiment, for each separate and distinct region in a digitallayout image, a size of a markup unit is set. Also, a size of anenclosing boundary or a grid element may be set based on a resolution ofa manufacturing process and a size of the resulting physical product.For example, if the resulting product is a small product, such as a caseof a cell phone, a grid size may be set to 1 cm; however, if theresulting product is a large product, such as a shirt, then a grid sizemay be set to 2.5 cm, or so. The size of the grid may be adjusted ormodified if the initial size of the grid is too small or too large.

Furthermore, for each separate and distinct region in a digital layoutimage, an offset or border of a markup is set within a grid element.

In an embodiment, for each separate and distinct region in a digitallayout image, a color of the color of the markup. A color may be setbased on the instructions provided by a manufacturer or based on atechnique used to detect the markup during the manufacturing processand/or by a system that will be used to process a photograph of aresulting physical product. The color may be also set based on thecapabilities of the processing system so that the markup can be removedfrom the photograph of the resulting physical product. In oneembodiment, the color is set to yellow, and the markup is printed on theresulting physical product at 30% yellow for a color segmentation.

In an embodiment, a width and a height of the grid is set. The width andthe height may be set in such a way so that the grid can cover theentire region of a digital layout image to be marked using a digitalmarkup image.

In an embodiment, a grid is generated according to the setting describedabove. The grid may be placed or overlaid over a digital layout image.The grid may be simply projected onto a digital layout image or rotatedor wrapped in such a way that there is a continuity of the grid acrossthe corresponding seams joining the respective regions. An example of adigital marker image overlaid over a digital layout image is describedin FIG. 5 .

FIG. 5 illustrates an example of overlaying a digital markup image overa digital layout image. The depicted example depicts two digital layoutimages: a digital layout image 430 and a digital layout image 450. Bothimages represent physical cut-out forms that may be used by amanufacturer, or a seamstress, to cut out pieces of fabric to assemble,for example, a pair of leggings.

Both digital layout image 430 and digital layout image 450 have threetypes of markings. One marking, depicted using a broken line 410,indicates a cut-out line. This line is to be followed to cut a fabric.Another marking, depicted using a solid line 420, indicates a stitchline which is to be followed by a sewing machine to assemble thegarment. Other marking, depicted using a dashed line 440, indicates asafe line beyond which neither cutting nor stitching should beperformed.

In FIG. 5 , digital layout image 430 is used to illustrate overlying 550a digital markup image over a layout image. As shown in FIG. 5 , a gridis placed onto digital layout image 430, and a digital markup image isprojected, or overlaid, onto the grid of digital layout image 430. Inthis example, the grid is stretched along the widest portion of digitallayout image 430. In other examples, the grid may be rotated, wrapped,or otherwise deformed to allow for continuity across seams that would beused to join individual pieces of fabric to assemble a physical product.

In the example depicted in FIG. 5 , a digital markup image does notextend to the entire perimeter of a digital layout image. This may berecommended to preserve a scale of the design. For example, if thedigital layout image represents a cut-out form of a pair of leggings,then a scale of the markup pattern in for example, the verticaldirections of the leggings may be preserved. If the design layout imagerepresents a cut-out form of a fitted shirt, then a scale of the designin for example, a horizontal direction of the shirt may be preserved.

A grid-based approach for overlaying a digital marker image over adigital layout image while preserving a scale of the design in at leastone direction may be appropriate for various products in which depictingthe design in the correct scale matters. For these products, thisapproach may be more appropriate than, for example, fitting the patternto the exact edge. If a uniform scale is more important than the abilityto tile a pattern across the edge, then fitting the pattern to the exactedge may be undesirable. Referring again to the example of a pair ofleggings, it may be desirable to map a digital markup image onto adigital layout image in such a way so that at least some patterns willappear continuously across both legs.

FIG. 6 illustrates an example design area that shows a digital markupimage overlaid over a digital layout image and that is displayed to auser in a graphical user interface. The example depicts the design areafor a pair of leggings. To illustrate an example, a lower portion of thelegging has been cropped.

In the depicted example, a digital layout image represents a cut-outform that has three types of markings. One marking, depicted using abroken line 610, indicates a cut-out line. This line is to be followedto cut a fabric. Another marking, depicted using a solid line 620,indicates a stitch line which is to be followed by a sewing machine toassemble the garment. Other marking, depicted using a dashed line 630,indicates a safe line beyond which neither cutting nor stitching shouldbe performed.

As it may be gleaned from FIG. 6 , a digital markup image is overlaidover a digital layout image in such a way that the pattern of thedigital markup overlaps itself in the center of the example design area.When the pattern is printed on the fabric, it is masked by the cutlines.

IV. Generating Manufacturing Files Containing Digital Markups

In step 1240, one or more manufacturing files are generated based on adigital markup layout. A digital markup layout may be a digital imageobtained by overlaying a digital markup image over a digital layoutimage. If the digital markup layout is obtained by overlaying thedigital markup image over the digital layout image, then the digitalmarkup layout includes at least a portion of the digital markupimprinted on the digital layout.

Manufacturing files are digital files that provide at least a digitalmarkup layout. Manufacturing files may also include instructions thatspecify the details for cutting physical components out of fabric, orother material, and the details for processing the components toassemble the components into a physical reference product. Manufacturingfiles may be represented in any format that a computer system of amanufacturer can accept and process.

Manufacturing files may be generated based on a digital markup layoutand based on settings used by a software application used to overlay adigital markup image over a digital layout image. This may be performedautomatically by selecting one or more settings from a graphical userinterface displayed by a software application used to generate a digitalmarkup layout. This may also be performed manually by having a userselect a digital markup layout and embedding instructions for processingthe digital markup layout. For example, using a graphical userinterface, a user may select a digital markup layout, and requestgenerating a manufacturing file based on the digital markup layout.

In an embodiment, one or more manufacturing files are transmitted to acomputer system of a manufacturer. The files may be transmitted via anytype of communications connection and any type of communicationsnetwork.

Upon receiving one or more manufacturing files, an employee of amanufacturer may print out a digital markup layout on a substrate. Forexample, if the digital markup layout is intended for manufacturing apiece of clothing, then the digital markup layout may be printed on acloth or a fabric.

Once a digital markup layout is printed on a substrate, the substratemay be cut into pieces that are to be used to assemble a physicalreference product. The cutting may be performed manually orautomatically. For example, an employee may use the digital markuplayout printed on a cloth, and cut the cloth along the cutting lines,such as line 610 in FIG. 6 .

If instructions for cutting a substrate are included in a manufacturingfile provided to a manufacturer, then the instructions may be used byeither an employee or by a computer system of the manufacturer toperform the cutting to obtain one or more components of a physicalreference product.

Once one or more components of a physical reference product are cut, orotherwise obtained, the components are assembled into the physicalreference product. For example, the pieces may be sewn together to forma pair of leggings. The pair of leggings will have a digital markupimprinted on the leggings.

In step 1250, a test is performed to determine whether a physicalreference product has been received from a manufacturer or a shipper. Ifthe physical reference product has been received, then step 1260 isperformed. Otherwise, the test in step 1250 is repeated.

In some situations, a physical reference product is not received from amanufacturer because one or more manufacturing files generated based ona digital markup layout were corrupted or had some errors. In suchsituations, steps 1230-1250 may be repeated so that new manufacturingfiles are generated, and a physical reference product is created andprovided.

V. Generating a Digital Reference Image

In an embodiment, a reference product is received from a manufacturer ora shipper facilitating delivery of the reference product. A referenceproduct is typically a physical product manufactured based onmanufacturing files provided to a manufacturer.

In step 1260, a digital reference image is generated based on areference product. A digital reference image may be a photograph of areference product. Depending on the nature of the reference product, theproduct may be featured on a model, a mannequin, a stand, or in anyother arrangements appropriate for demonstrating the reference product.For example, if a reference product is a pair of leggings, then theleggings may be put on a mannequin, and a photograph of the mannequinwearing the leggings may be taken. If a reference product is a piece offurniture with a cloth upholstery, then the piece of upholsteredfurniture may be featured in a showroom or in a private house.

In an embodiment, a reference product may be provided with instructionsspecifying how the reference product is to be featured, lit,illuminated, or otherwise processed. The instructions may be encoded ina code provided along with the reference product. The instructions mayalso be obtained from a server, a cloud system or other computingsystem.

Instructions that accompany a reference product may specify a rig to beused to photograph the reference product. The instructions may alsospecify the method that was used to process digital process markupsimprinted in the reference product. Furthermore, the instructions mayinclude suggestions for placing, illuminating, and arranging thereference product to produce a high quality photograph of the referenceproduct. The instructions may also specify a position of the camera, aposition of the light sources illuminating the reference product, andlocations of other elements that may enhance the quality of thephotograph of the reference product.

In an embodiment, a reference product is placed in an automatedphotographic system. The manner in which the reference product is placedin the system may be specified by instructions associated with thereference product, or by a photographer or a designer. For example, thereference product may be placed on a stand, a fixture, or a mountingconfigured to support the reference product.

In an embodiment, a reference product placed in an automatedphotographic system is illuminated using one or more light sources orany type of lighting. The type of lighting and the positioning of thelighting may be specified in the instructions that accompany thereference product, by a photographer or by a designer.

In an embodiment, one or more photographs are taken of a referenceproduct. A photograph may be taken using any type of camera. Forexample, the photograph may be taken using a digital camera which isconfigured to capture digital images. The photograph may also be takenusing an analog camera. In this case, a photograph taken using an analogcamera may be digitally scanned, and a digital image may be generatedfrom the photograph. The obtained digital images are referred to asdigital reference images and may be stored in a storage device.

FIG. 7 illustrates an example digital reference image. The exampledepicts a digital reference image of a pair of leggings 710 featured ona model 720. Pair of leggings 710 is depicted in such a way that thefront and the back of the leggings wrap around model 720, and a leg ofthe leggings wraps around a leg of model 720, as depicted using element730.

VI. Segmenting Digital Markups in Digital Reference Images

In step 1270, a digital reference image is digitally processed tosegment any markup in the digital reference image. Segmenting a markupin an image is also referred to as recognizing the markup in the digitalreference image.

In an embodiment, a digital reference image is analyzed using anelectronic device to recognize markups depicted in the image.Recognizing markups in a digital reference image presents manychallenges. For example, the markups may not be clearly visible in adigital reference image. Furthermore, it may be difficult to distinguishan edge or an element of a markup in a region in which seams of thereference product are depicted, or in a region in which the referenceproduct wraps around a mannequin and thus it is only partially visibleon the digital reference image.

Also, it may be difficult to recognize a markup in a digital referenceimage if the image has a low contrast between the markup imprinted inthe digital reference image and regions that do not include markups.Moreover, it may be difficult to recognize a markup in a digitalreference image if the image has a poor quality, a low resolution, a lowcontrast, or a low brightness. In addition, a digital reference imagemay include so many different elements that recognizing a markup in theimage may be quite difficult. Furthermore, a digital reference image mayinclude not one, but a plurality of markups. Therefore, the process ofrecognizing a markup in a digital reference image may be sometimesrepeated several times until one or more markups are segmented in theimage.

A. Segmenting a Digital Markup Using a Color-Filtering Approach

FIG. 8 illustrates an example digital reference image to be processedusing a color filtering approach to determine a digital markup imprintedin the image. The example depicts a digital reference image of a pair ofleggings 810 featured on a model 820. Pair of leggings 810 is depictedin such a way that the front and the back of the leggings wrap aroundmodel 820, and each leg of the leggings wraps around the model's leg.The example depicts that the leggings wrap around a right leg of model820, as depicted using an element 830.

In an embodiment, a digital reference image is processed to determine aplurality of distinct regions in the digital layout image. The distinctregions may be determined in the digital layout image by analyzing colorvalues for the digital layout image and determining the distinct regionsthat include the color values that are different from the color valuesof other regions. For example, if a digital reference image depicts apair of white leggings 820 with black (or yellow) printing on it, thenthe digital reference image may be processed to identify a plurality ofdistinct regions that include the back (or yellow) printing, and not thewhite regions.

In an embodiment, a digital reference image is processed to determine aplurality of distinct regions in the digital layout image based on colorand color differences in the digital reference image. Using thisapproach, a product image may be partitioned into different regionsbased in part on color differences. For example, each location of theproduct markup image input image may be assigned a color differencevalue indicating how different the location is from neighboringlocations in terms of color. Each location may be assigned an imageregion of the plurality of image regions, and locations may beconsidered for region assignments according to an order that is based,at least in part, on the color difference value assigned to the pixel.For example, locations with low associated color difference values maybe assigned regions before locations with high associated colordifference values. This approach may be referred to as a color shedapproach.

In an embodiment, region information determined based on visualizationof a customizable product in a digital reference image allowsdetermining a position of particular markup portions in the digitalreference image. Based on the determined positions, an image processingsystem may determine instructions for rendering computer-generatedvisualization of the customized products. The region information mayalso be utilized to verify the quality of a customized product after thecustomized product has been manufactured.

In an embodiment, a digital reference image may be partitioned into aplurality of image regions. The partitioning processes may result inpartitioning at least the markup portion of the digital reference imageinto a set of regions, where each region of the set of regionsrepresents a portion of the markup.

In an embodiment, an image partitioning process may result indetermining regions information that identifies, for example, for eachfound region, the image locations that belong to the region. Theinformation may also include descriptions of the area of each region,and/or the representative color of the region. In some embodiments, eachimage location corresponds to a separate pixel. In other embodiments,each image location is a group of pixels, or a markup element.

In an embodiment, the resulting region information is utilized in theautomatic visualization of a custom product, which may be customizedaccording to customer-provided parameters.

FIG. 13 illustrates an example process for identifying markups indigital reference images using a color filtering approach.

In step 1310, a digital image comprising a markup is received. Thedigital image may correspond to a digital reference image describedabove.

In step 1320, one or more locations in the digital image are identified.A location identified in the digital image may be a group of pixels orjust a pixel. Hence, a location may correspond to either a group ofpixels or an individual pixel.

In step 1330, one or more neighboring locations that neighbor to aparticular location in a digital image are identified. If locationsinclude groups of pixels, then for a particular location, one or moreneighboring locations adjacent to the particular location include groupsof pixels. If locations include individual pixels, then for a particularlocation, one or more neighboring pixels adjacent to the particularlocation are just individual pixels.

In step 1340, a color difference value for each of the locations isdetermined. For example, for a particular location, a particular colorvalue for the particular location and color values for the neighboringlocations are determined, and an average value of differences betweenthe particular color values and the neighboring color values isdetermined. The color difference value may also be determined usingother approaches.

The process of determining a color difference value for each location isrepeated for each location identified in the digital image.

If in step 1350, if it is determined that the process has been repeatedfor all locations identified in the digital image, then step 1360 isperformed. Otherwise, steps 1330-1340 are repeated for the remaininglocations.

In step 1360, the locations identified in the digital image are orderedaccording to the color difference values determined for the locations.

In step 1370, the digital image is partitioned based on the orderedlocations into one or more regions. The partitioning includes assigning,to each location, an image region of the plurality of image regions. Theassignment between the locations and regions may be performed accordingto an order that is based, at least in part, on the color differencevalue assigned to the pixel. For example, locations with low associatedcolor difference values may be assigned regions before locations withhigh associated color difference values.

By assigning regions to locations based on respective color differencevalues determined for the locations, the locations with low colordifference values (i.e., the locations that are very similar to theirsurrounding locations) may be more likely to be at the center of thedetermined regions or in other non-border locations of the regions. Suchan image partitioning approach may cause the resulting partition imageto be partitioned into regions that are more uniformly-colored thanregions determined according to other approaches.

In step 1380, based on the partitioning, data is identified thatrepresents at least a portion of a markup in the digital image.

It is possible that only a portion of the markup, not the entire markup,is identified in a digital reference image. This may happen because somemarkups may not be clearly visible in the digital reference image.Furthermore, this may happen because it may be difficult to distinguishan edge or an element of a markup in a region in which seams of thereference product are depicted. This may also happen in a region inwhich, for example, the reference product wraps around a mannequin andthus it is only partially visible on the digital reference image.

In step 1390, a particular markup is identified based on the datarepresenting at least a portion of the markup. Different approaches maybe implemented to identify a particular markup based on the datarepresenting at least a portion of the markup using the datarepresenting at least a portion of the markup. One approach includesgenerating a mapping between found markup regions and regions identifiedof a digital markup image.

Other approaches for identifying at least a portion of the markupimprinted on a digital reference image may include determining adistance value between locations. A distance value may be determined forexample, as a difference between the color of a pixel in a digitalreference image and the color of a digital markup image in a particularcolor space. The distance values may be computed for each pixel in adigital reference image, and a region may be created based on a group ofpixels for which the respective color differences are the smallest.

Another approach for determining a markup region in a digital referenceimage may use a color spatial frequency approach. This approach mayinclude computing a bandpass determined based on the dot size in adigital markup image and comparing the bandpass with the characteristicsof the pixels of the digital reference image.

Referring again to FIG. 12 , in step 1280, one or more found markupregions are identified in a digital reference image. The found markupregions are the regions that most likely, or with some certainty,include at least a portion of a markup.

B. Building a List of Markup Identifiers

In an embodiment, one or more found markup regions are used to build alist of MarkupIDs of markups potentially included in the found markupregions.

In an embodiment, for each found markup region, of one or more foundmarkup regions identified in a digital reference image, a list ofMarkupIDs is generated. A list of MarkupIDs for a particular foundmarkup region identified in a digital reference image is a list thatincludes identifiers of markups that are included in the particularfound markup region of the digital reference image or may be included inthe particular found markup region of the digital reference image. Forexample, a particular found markup region may include one or moremarkups, one or more portions of one or more markups, one or moreportions of one markup and a depiction of another markup, and so forth.

A list of MarkupIDs generated for a particular found markup region maybe any type of data structure, such as a data table, a data index, adata space region indexed using pointers, and so forth.

In an embodiment, a list generated for a particular found markup regioncomprises a MarkupID of a particular digital markup image, a positioninformation of the particular found markup region within a digitalreference image, and any additional metadata that may be used toidentify the particular digital markup image. A list for a particularfound markup region may include more than one MarkupID if more than onedigital markup image may be included in the particular found markupregion.

A list comprising MarkupIDs may be generated for a particular foundmarkup region if the particular found markup region meets certainconditions. For example, a test may be performed to determine whether aparticular found markup region is a valid region. The test may includechecking the size of the dots included in the found markup region, thesize of the areas between the dots, and the like.

A particular MarkupID may be included in a list for a particular foundmarkup region if the particular MarkupID meets certain criteria. Forexample, a test may be performed to determine whether a checksum of theparticular MarkupID is valid. As described below, a checksum of a markupmay be defined by a count of dots in the vertical direction and/or acount of dots in the horizontal directions. For example, if the markupis expected to have five dots horizontally and five dots vertically,then a checksum of the markup is 5×5. Testing the checksum of the markupmay involve testing whether the markup has the checksum of 5×5.

A particular MarkupID may be included in a list for a particular foundmarkup region if the particular MarkupID corresponds to a markup thathas acceptable extents defined in terms of dots in a markup pattern ofthe markup. This may include rotations of the markup elements within themarkup pattern. Based on extents, a markup transformation for the markupmay be generated and used to determine and verify a map of the markupdots.

A particular MarkupID may be included in a list for a particular foundmarkup region if a digital signature of a corresponding markup isacceptable or correct. A digital signature of a markup may be defined asa count of regions distinguishable in the markup. For example, if amarkup includes 16 tiles of markup elements, then a digital signature ofthat markup may be 16. Testing a digital signature of a markup mayinclude determining whether the markup indeed includes 16 tiles.

Generating a list of MarkupIDs for a particular found markup region mayinclude determining a rotation for a markup associated with a MarkupID.Some markups may be expressed using one or more rotation parameters. Anexample of such a markup is described in FIG. 11 .

In an embodiment, additional information about a markup may be obtainedor generated before a MarkupID of the markup is included in a list ofMarkupIDs for a found markup region. For example, a rotation informationof the marker may be obtained and included in the list. Furthermore,surface derivatives may be computed based on a spacing of dot centerswithin the found markup region. Moreover, the corrected markup regiongeometry may be calculated using the information about the markuptransformation and the information about the surface derivatives andincluded in the list.

In step 1290, one or more reference markup regions identified in adigital markup image are mapped onto one or more found markup regionsidentified in a digital reference image. The resulting mapping may bereferred to as a region mapping.

In an embodiment, one or more reference markup regions are identified ina digital markup image. This may be performed before a digital markuplayout is sent to a manufacturer, or when a digital markup image isgenerated and stored.

In an embodiment, one or more reference markup regions are identified ina digital markup image by dividing the digital markup image into one ormore areas corresponding to individual markup elements. For example, ifa markup includes four markup elements repeated horizontally and fourmarkup elements repeated vertically, then the markup may be divided into16 areas, which may be equated to one or more reference markup regions.

For each reference markup region, of one or more reference regions, aposition of the reference markup region within a digital reference imageand extents of the reference markup region may be identified.

In an embodiment, a list of reference markup regions for a digitalmarkup image is created. A list of reference markup regions may includeinformation about the regions identified within the digital markupimage. For each reference markup region identified in a digital markupimage, a list may include a MarkupID of the markup, information aboutthe position and extents of the markup region.

Once lists of reference markup regions for a digital markup image arecreated, the regions of the markups may be mapped onto one or more foundmarkup regions identified in a digital reference image. The mapping maybe performed by traversing the lists and trying to match thecharacteristics of the reference markup region with the characteristicsof found markup regions. The traversing and matching may be acceleratedby using for example, data hashing techniques.

C. Generating a Geometry Mapping

In step 1292, a geometry map is generated. A geometry map may begenerated based on a mapping between reference markup regions of markupsand one or more found reference regions identified in a digitalreference image.

In an embodiment, a geometry map is generated by building a set ofpolygons. A set of polygons may be determined based on reference markupregions identified in a markup and mapping the polygons onto one or morefound reference regions identified in a digital reference image. Eachvertex of a polygon may have an associated reference point in areference markup region and a found marker point in a found referenceregion. A set of polygons may be built using the information in a listof reference markup regions for the markups, extents information for themarkups, and adjacency information for the reference markup regions.

In an embodiment, a set of polygons determined for a geometry mapincludes information about the polygons that are adjacent to each other.The adjacency information for two polygons may be determined based on adistance between the centers of the polygons and based on a distancetolerance value. The geometry map may also include information about theaverage contributions of adjacent vertices of the neighboring polygonsto each of the polygons.

In an embodiment, once a geometry map for a digital reference image iscreated, markups, or portions of the markups, may be removed from thedigital reference image. Since the purpose of depicting a markup in adigital reference image is to register a reference points of the markupin the digital reference image, once a geometry map is created for thedigital reference markup, a further processing of the digital referencemarkup may be performed on an image in which the markups have beenremoved.

VII. Generating an Interactive Asset Image

In step 1294, an interactive asset is rendered from an unmarked digitalimage. This may include processing a digital reference image by removingany marking. An example of the process is described in FIG. 9 .

FIG. 9 illustrates an example image having markups removed. The exampledepicts a pair of leggings 910 featured on a model 920. In contrast to apair of leggings depicted in FIG. 8 , the leggings in FIG. 9 do not haveany imprinted marking. For example, if the leggings in FIG. 8 were whitewith a yellow imprinted marking, then the leggings in FIG. 9 show awhite pair of leggings with no yellow marking.

In FIG. 9 , a pair of leggings 910 is depicted in such a way that thefront and the back of the leggings wrap around model 920, and each legof the leggings wraps around the model's leg. The example depicts thatthe leggings wrap around a right leg of model 920, as depicted using anelement 930.

In an embodiment, an unmarked digital image is used as an interactiveasset image. The interactive asset image may have the same shape andsize characteristics and the unmarked digital image. Furthermore, theinteractive asset image may have the same background characteristics andthe unmarked digital image.

VIII. Generating a Customized Product Image

In an embodiment, an interactive asset image is further processed. Thefurther processing may include applying a user pattern to theinteractive asset image to generate a customized product image. Forexample, if an interactive asset image depicts a pair of leggings, theinteractive asset image may be interactively customized by allowing auser to apply a custom pattern to the leggings and generate an image ofthe customized product. The customization may allow for deforming thecustomized pattern in the customized product image in the same fashionas a digital markup image was deformed on some surfaces of the digitalreference image described above.

In step 1296, a user pattern is mapped onto an interactive assetaccording to a geometry mapping to generate a customized product image.A user pattern may be for example, a customized pattern generated by acustomer who designs the interactive asset using a website application.A user pattern may also be a customized pattern generated by a webdeveloper who designs different customization options that may beavailable to customers who access the website.

In an embodiment, a customization of an interactive asset image using auser pattern allows for deforming the user pattern in a customizedproduct image in the same fashion as a digital markup image was deformedon some surfaces of the digital reference image.

In an embodiment, a customization of an interactive asset image using auser pattern includes encoding means for rendering an unmarked digitalimage into an interactive asset, and encoding means for mapping a userpattern or image in a photo-real manner using a geometry map onto aninteractive asset image.

FIG. 10 illustrates an example customized product image that is renderedby applying a user pattern to an interactive asset image. The figuredepicts a pair of leggings 1010 modelled by a model 1030. Pair ofleggings 1010 has a user-defined pattern imprinted on the garment.Noticeably, the user pattern is deformed in the customized product imagein the same fashion as a digital markup image was deformed on thesurfaces of the digital reference image depicted in FIG. 7 . Inparticular, a pair of leggings 1010 is depicted in such a way that thefront and the back of the leggings wrap around model 1020, and a leg ofthe leggings wraps around a leg of model 1020, as depicted using element1030.

In an embodiment, an approach for customizing digital images provides anautomated method for realistically visualizing a customized image. Theapproach includes encoding a digital signature in markups used tomanufacture reference products.

In an embodiment, an approach includes processes that allow imprintingmarkups on a product manufactured with specified colors and patterns.The methods allow detecting specific marked-up regions in a digitalreference image, find the position and orientation of these regions,apply photo-realistic visualization of customized patterns to theproduct, and display digital images of the customized product.

The processes work well when at least a portion of the markups isvisible in a digital reference image. For example, the softwarealgorithm used for finding the markup in a digital reference image maybe fully automated when the markup is well-visible in a digitalreference image, or when only a small portion of the markup is visible.

In an embodiment, techniques for detecting and mapping digital markersonto a digital reference image may be implemented in a specializedprogram application. The application may be encoded using open sourcecomputer vision projects, such as the openCV.

In an embodiment, an approach for using digital markups to customizeproducts is implemented in an Augmented Reality system. The system maybe calibrated using a set of digital markup images and other calibrationtechniques. For example, a digital marker boundary frame may be used toprovide a coordinate frame that allows identifying the marker in animage. Markers used to calibrate the Augmented Reality system areusually presented on a flat or non-deformed surface to allow anefficient identification and mapping of the markup. In an embodiment,digital markers are encoded using various techniques, including QRcodes.

IX. Implementation Example—Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques or may include digital electronic devices such as one or moreapplication-specific integrated circuits (ASICs) or field programmablegate arrays (FPGAs) that are persistently programmed to perform thetechniques, or may include one or more general purpose hardwareprocessors programmed to perform the techniques pursuant to programinstructions in firmware, memory, other storage, or a combination. Suchspecial-purpose computing devices may also combine custom hard-wiredlogic, ASICs, or FPGAs with custom programming to accomplish thetechniques. The special-purpose computing devices may be desktopcomputer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 14 is a block diagram that illustrates a computersystem 1400 upon which an embodiment of the invention may beimplemented. Computer system 1400 includes a bus 1402 or othercommunication mechanism for communicating information, and a hardwareprocessor 1404 coupled with bus 1402 for processing information.Hardware processor 1404 may be, for example, a general purposemicroprocessor.

Computer system 1400 also includes a main memory 1406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1402for storing information and instructions to be executed by processor1404. Main memory 1406 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1404. Such instructions, when stored innon-transitory storage media accessible to processor 1404, rendercomputer system 1400 into a special-purpose machine that is customizedto perform the operations specified in the instructions.

Computer system 1400 further includes a read only memory (ROM) 1408 orother static storage device coupled to bus 1402 for storing staticinformation and instructions for processor 1404. A storage device 1410,such as a magnetic disk, optical disk, or solid-state drive is providedand coupled to bus 1402 for storing information and instructions.

Computer system 1400 may be coupled via bus 1402 to a display 1412, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1414, including alphanumeric and other keys, iscoupled to bus 1402 for communicating information and command selectionsto processor 1404. Another type of user input device is cursor control1416, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1404 and for controlling cursor movement on display 1412. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1400 in response to processor 1404 executing one or moresequences of one or more instructions contained in main memory 1406.Such instructions may be read into main memory 1406 from another storagemedium, such as storage device 1410. Execution of the sequences ofinstructions contained in main memory 1406 causes processor 1404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical disks, magnetic disks, or solid-state drives, suchas storage device 1410. Volatile media includes dynamic memory, such asmain memory 1406. Common forms of storage media include, for example, afloppy disk, a flexible disk, hard disk, solid-state drive, magnetictape, or any other magnetic data storage medium, a CD-ROM, any otheroptical data storage medium, any physical medium with patterns of holes,a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1404 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1402. Bus 1402 carries the data tomain memory 1406, from which processor 1404 retrieves and executes theinstructions. The instructions received by main memory 1406 mayoptionally be stored on storage device 1410 either before or afterexecution by processor 1404.

Computer system 1400 also includes a communication interface 1418coupled to bus 1402. Communication interface 1418 provides a two-waydata communication coupling to a network link 1420 that is connected toa local network 1422. For example, communication interface 1418 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1418 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1418 sends and receives electrical, electromagnetic, oroptical signals that carry digital data streams representing varioustypes of information.

Network link 1420 typically provides data communication through one ormore networks to other data devices. For example, network link 1420 mayprovide a connection through local network 1422 to a host computer 1424or to data equipment operated by an Internet Service Provider (ISP)1426. ISP 1426 in turn provides data communication services through theworldwide packet data communication network now commonly referred to asthe “Internet” 1428. Local network 1422 and Internet 1428 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1420 and through communication interface 1418, which carrythe digital data to and from computer system 1400, are example forms oftransmission media.

Computer system 1400 can send messages and receive data, includingprogram code, through the network(s), network link 1420 andcommunication interface 1418. In the Internet example, a server 1430might transmit a requested code for an application program throughInternet 1428, ISP 1426, local network 1422 and communication interface1418.

The received code may be executed by processor 1404 as it is received,and/or stored in storage device 1410, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A computer implemented method comprising:receiving, at an electronic device, a digital reference image of areference product that has been manufactured based on, at least in part,one or more manufacturing files; segmenting the digital reference imageinto one or more found markup regions by performing: applying a colorshed to the digital reference image, based on the color shed applied tothe digital reference image, determining distance values between colorvalues in areas of interest in the digital reference image and colorvalues in a reference photograph, generating, based on the distancevalues, a spatial frequency histogram for the digital reference image;based on the spatial frequency histogram for the digital referenceimage, partitioning the digital reference image into one or more imageregions; based on the partitioning and the one or more image regions,determining data representing at least one digital markup image withinthe digital reference image; and assigning a markup identifier to the atleast one digital markup image identified in the digital referenceimage.
 2. The method of claim 1, further comprising: determining one ormore reference markup regions within the digital markup image that isassociated with the markup identifier; receiving, at the electronicdevice, a digital layout image that represents a form of a product usedto manufacture a reference product; generating, using the electronicdevice, a digital markup layout by overlaying the digital markup imageover the digital layout image; based on, at least in part, the digitalmarkup layout, generating the one or more manufacturing files comprisingdigital data usable in manufacturing the reference product; identifyingone or more found markup regions in the digital reference image;determining a region mapping that maps one or more of the one or morereference markup regions to the one or more found markup regions; basedon, at least in part, the region mapping, generating a geometry map;generating an interactive asset image by removing the one or more foundmarkup regions from the digital reference image; based on, at least inpart, the region mapping and the geometry map, generating a customizedproduct image by applying a user pattern to the interactive asset image;and causing the customized product image to be displayed on a displaydevice.
 3. The method of claim 2, further comprising: determining achecksum for the digital markup image; generating a digital patternimage that has the checksum embedded in the digital pattern image. 4.The method of claim 2, further comprising: generating the markupidentifier based on contents of the digital markup image; storing thedigital markup image in an association with the markup identifier in amarkup index data structure.
 5. The method of claim 2, furthercomprising: determining a plurality of distinct regions in the digitallayout image; for each distinct region of the plurality of distinctregions, overlaying the digital markup image onto a distinct region. 6.The method of claim 2, wherein the geometry map comprises a plurality ofpolygons.
 7. The method of claim 2, wherein geometry map maps, for theone or more found markup regions, include a reference point of aparticular found markup region corresponding to a reference point of aparticular reference markup region.
 8. The method of claim 6, whereineach polygon of the plurality of polygons includes a plurality ofvertices describing a relation between a particular found markup regionand a particular reference markup region.
 9. The method of claim 1,wherein a color difference value is assigned to each location of aplurality of locations within the digital reference image; wherein aparticular color difference value assigned to a particular location isdetermined based in part on the digital reference image and indicateshow different a color of the particular location is from colors of otherlocations neighboring the particular location.
 10. The method of claim2, further comprising: generating a list of markups for the one or morefound markup regions.
 11. One or more non-transitory computer-readablestorage media, storing one or more computer instructions which, whenexecuted by one or more processors, cause the one or more processors toperform: receiving, at an electronic device, a digital reference imageof a reference product that has been manufactured based on, at least inpart, one or more manufacturing files; segmenting the digital referenceimage into one or more found markup regions by performing: applying acolor shed to the digital reference image, based on the color shedapplied to the digital reference image, determining distance valuesbetween color values in areas of interest in the digital reference imageand color values in a reference photograph, generating, based on thedistance values, a spatial frequency histogram for the digital referenceimage; based on the spatial frequency histogram for the digitalreference image, partitioning the digital reference image into one ormore image regions; based on the partitioning and the one or more imageregions, determining data representing at least one digital markup imagewithin the digital reference image; and assigning a markup identifier tothe at least one digital markup image identified in the digitalreference image.
 12. The one or more non-transitory computer-readablestorage media of claim 11, storing additional instructions for:determining one or more reference markup regions within the digitalmarkup image that is associated with the markup identifier; receiving,at the electronic device, a digital layout image that represents a formof a product used to manufacture a reference product; generating, usingthe electronic device, a digital markup layout by overlaying the digitalmarkup image over the digital layout image; based on, at least in part,the digital markup layout, generating the one or more manufacturingfiles comprising digital data usable in manufacturing the referenceproduct; identifying one or more found markup regions in the digitalreference image; determining a region mapping that maps one or more ofthe one or more reference markup regions to the one or more found markupregions; based on, at least in part, the region mapping, generating ageometry map; generating an interactive asset image by removing the oneor more found markup regions from the digital reference image; based on,at least in part, the region mapping and the geometry map, generating acustomized product image by applying a user pattern to the interactiveasset image; and causing the customized product image to be displayed ona display device.
 13. The one or more non-transitory computer-readablestorage media of claim 12, storing additional instructions for:determining a checksum for the digital markup image; generating adigital pattern image that has the checksum embedded in the digitalpattern image.
 14. The one or more non-transitory computer-readablestorage media of claim 12, storing additional instructions for:generating the markup identifier based on contents of the digital markupimage; storing the digital markup image in an association with themarkup identifier in a markup index data structure.
 15. The one or morenon-transitory computer-readable storage media of claim 12, storingadditional instructions for: determining a plurality of distinct regionsin the digital layout image; for each distinct region of the pluralityof distinct regions, overlaying the digital markup image onto a distinctregion.
 16. The one or more non-transitory computer-readable storagemedia of claim 12, wherein the geometry map comprises a plurality ofpolygons.
 17. The one or more non-transitory computer-readable storagemedia of claim 12, wherein geometry map maps, for the one or more foundmarkup regions, include a reference point of a particular found markupregion corresponding to a reference point of a particular referencemarkup region.
 18. The one or more non-transitory computer-readablestorage media of claim 16, wherein each polygon of the plurality ofpolygons includes a plurality of vertices describing a relation betweena particular found markup region and a particular reference markupregion.
 19. The one or more non-transitory computer-readable storagemedia of claim 11, wherein a color difference value is assigned to eachlocation of a plurality of locations within the digital reference image;wherein a particular color difference value assigned to a particularlocation is determined based in part on the digital reference image andindicates how different a color of the particular location is fromcolors of other locations neighboring the particular location.
 20. Theone or more non-transitory computer-readable storage media of claim 12,storing additional instructions for: generating a list of markups forthe one or more found markup regions.